System and method for building decision tree classifiers using bitmap techniques

ABSTRACT

A method, system, and computer program product for counting predictor-target pairs for a decision tree model provides the capability to generate count tables that is quicker and more efficient than previous techniques. A method of counting predictor-target pairs for a decision tree model, the decision tree model based on data stored in a database, the data comprising a plurality of rows of data, at least one predictor and at least one target, comprises generating a bitmap for each split node of data stored in a database system by intersecting a parent node bitmap and a bitmap of a predictor that satisfies a condition of the node, intersecting each split node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps, and counting bits of each intersected bitmap to generate a count of predictor-target pairs.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method, system, and computer programproduct for counting predictor-target pairs for a decision tree modelthat provides the capability to generate count tables that is quickerand more efficient than previous techniques.

2. Description of the Related Art

Data mining is a technique by which hidden patterns may be found in agroup of data. True data mining doesn't just change the presentation ofdata, but actually discovers previously unknown relationships among thedata. Data mining is typically implemented as software in, or inassociation with, database systems. Data mining includes several majorsteps. First, data mining models are generated by one or more dataanalysis algorithms. Initially, the models are “untrained”, but are“trained” by processing training data and generating information thatdefines the model. The generated information is then deployed for use indata mining, for example, by providing predictions of future behaviorbased on specific past behavior.

One important form of data mining model is the decision tree. Decisiontrees are an efficient form for representing decision processes forclassifying entities into categories or constructing piecewise constantfunctions in nonlinear regression. A tree functions in a hierarchicalarrangement; data flowing “down” a tree encounters one decision at atime until a terminal node is reached. A particular variable enters thecalculation only when it is required at a particular decision node andonly one variable is used at each decision node.

Classification is a well-known and extensively researched problem in therealm of Data Mining. It has found diverse applications in areas oftargeted marketing, customer segmentation, fraud detection, and medicaldiagnosis among others. Among the methods proposed, decision trees arepopular for modeling data for classification purposes. The primary goalof classification methods is to learn the relationship between a targetattribute and many predictor attributes in the data. Given instances(records) of data where the predictors and targets are known, themodeling process attempts to glean any relationships between thepredictor and target attributes. Subsequently, the model is used toprovide a prediction of the target attribute for data instances wherethe target value is unknown and some or all of the predictors areavailable.

Classification using decision trees is a well-known technique that hasbeen around for a long time. However, the early decision tree algorithmsworked well only on small amounts of data and did not scale to largedatasets. Most of the well known algorithms for building decision trees,like SLIQ, SPRINT, RainForest, BOAT etc., construct count tables to findsplitting attributes and split points. Count tables store record countsfor every (predictor value, target value) pairs at every node in thetree. As the build process goes deeper in the tree, constructing thesecount tables becomes very expensive in terms of computing resources andtime. A need arises for a technique by which such counting can beperformed more quickly and efficiently.

SUMMARY OF THE INVENTION

The present invention provides the capability to generate count tablesthat is quicker and more efficient than previous techniques.

In one embodiment of the present invention, a computer-implementedmethod of counting predictor-target pairs for a decision tree model, thedecision tree model based on data stored in a database, the datacomprising a plurality of rows of data, at least one predictor and atleast one target, the method comprises generating a bitmap for each treenode corresponding to a subset of data stored in a database system byintersecting a parent node bitmap and a bitmap of a predictor thatsatisfies a condition of the node, intersecting each split node bitmapwith each predictor bitmap and with each target bitmap to formintersected bitmaps, and counting bits of each intersected bitmap togenerate a count of per-node predictor-target pairs.

In one aspect of the present invention, each split node bitmap may beintersected with each predictor bitmap and with each target bitmap toform intersected bitmaps by intersecting each target bitmap with eachsplit node bitmap to form a plurality of intermediate bitmaps andintersecting each intermediate bitmap with each predictor bitmap to forman intersected bitmap. The target bitmaps and the split node bitmaps mayfit in a memory of a computer.

In one aspect of the present invention, each split node bitmap may beintersected with each predictor bitmap and with each target bitmap toform intersected bitmaps by, for each of a plurality of portions of thesplit node bitmaps, intersecting each target bitmap with each split nodebitmap in the portion of the split node bitmaps to form a plurality ofintermediate bitmaps and intersecting each intermediate bitmap with eachpredictor bitmap to form the intersected bitmaps. The target bitmaps anda portion of the split node bitmaps fit in a memory of a computer.

In one embodiment of the present invention, a computer-implementedmethod of generating a decision tree model comprises generating aplurality of bitmaps in the database system, the bitmaps generated fromdata stored in a database table in the database system, the databasetable comprising a plurality of rows of data, the plurality of bitmapscomprising a bitmap for each unique value of each predictor and targetand indicating whether or not that unique value of each predictor andtarget is present in each row of the database table, intersecting eachsplit node bitmap with each predictor bitmap and with each target bitmapto form intersected bitmaps, counting bits of each intersected bitmap togenerate a count of predictor-target pairs, determining a splitter valuefor the data in the database table using the counts of thepredictor-target pairs so as to split the data in the database tableinto a plurality of child nodes, each child node comprising a portion ofthe data in the database table, generating child bitmaps for the data ineach child node, recursively generating a bitmap for each child node byintersecting a parent node bitmap and a bitmap of a predictor thatsatisfies a condition of the child node, intersecting each child nodebitmap with each predictor bitmap and with each target bitmap to formintersected bitmaps, and counting bits of each intersected bitmap togenerate a count of predictor-target pairs, whereby a decision treemodel is formed.

In one aspect of the present invention, the bitmaps may be sorted bypredictor and predictor value and target and target value. The bitmapthat satisfies the condition of a particular node may be generated by,if the node is a root node, ORing each bitmap for each value of thepredictor, to form a single bitmap for all values of the predictor, and,if the node is below the root node, generating a bitmap by ORing eachbitmap for each value of the predictor that satisfies a condition of thenode split, to form a single bitmap for all values of the predictor thatsatisfy the condition of the node split and ANDing the single bitmapwith a bitmap for a node above the node. The bitmaps may be sorted bypredictor and predictor value and target and target value.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and advantages of the invention can be ascertained fromthe following detailed description that is provided in connection withthe drawings described below:

FIG. 1 illustrates an example of the application of a decision treemodel.

FIG. 2 is an exemplary data flow diagram of a process of building adecision tree model.

FIG. 3 is an exemplary flow diagram of a process of in-database buildingof a decision tree model.

FIG. 4 is an exemplary illustration of construction of bitmaps from rowsof data.

FIG. 5 is an example of an interface defining an SQL statement thatinvokes in-database generation of a decision tree model.

FIG. 6 is an example of the use of an SQL statement, such as thatdefined in FIG. 5, which invokes in-database generation of a decisiontree model.

FIG. 7 is an example of a PL/SQL API through which an SQL statement,such as that shown in FIG. 6, is invoked.

FIG. 8 is an exemplary block diagram of a database system, in which thepresent invention may be implemented.

FIG. 9 is an exemplary flow diagram of additional processing relating tocounting of predictor-target pairs and generation of count tablesincluding these counts.

FIG. 10 illustrates an example of processing where all the bitmaps andcount tables fit in memory.

FIG. 11 illustrates an example where only a portion of the predictorbitmaps fit in memory, and all of the target and node bitmaps fit inmemory

FIG. 12 illustrates an example where a portion of the predictor and nodebitmaps do not fit in memory.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention provides the capability to generate count tablesthat is quicker and more efficient than previous techniques. Thiscapability is particularly useful when implemented in an SQL tablefunction that encapsulates the concept of creating a decision tree basedon a dataset that is the input from a query. This table function takesthe input dataset along with some user-configurable information, and itdirectly produces a decision tree. The tree can then be used tounderstand the relationships in the data as well as to score newrecords.

The new table function is implemented inside the Relational DatabaseManagement System (RDBMS) by program code that supports this new SQLtable function. Integrating the process of building decision treesinside the RDBMS enables leveraging of many of the database's strengths,such as memory management, parallel execution, and recursive execution.Providing a simple SQL interface via a specialized table function makesthe integration of data mining into the database far simpler.

The SQL table function is an improvement over the use of standard SQL.It simplifies the query, but more importantly it simplifies the queryoptimization stages by making it explicit what type of operation isbeing processed. It enables the decision tree build process to leveragescalable, efficient, and robust database processing with a very simpleinterface.

Another advantage is that this method doesn't have to incur the expense,management, and security issues of moving the data to a specializedmining engine.

A decision tree is represented as a directed acyclic graph consisting oflinks and nodes. The structure defines a set of parent-childrelationships. Parent nodes contain splitting rules that define theconditions under which a specific child is chosen. The rules consist ofa splitting predictor, an operator, and one or more split values. Forexample, a rule might be IF AGE<=b 10 THEN Left Child ELSE Right Child.Another example is IF HAIR Color IN (Brown, Black) THEN Left Child ELSERight Child. In addition, each node can contain ancillary information,such as a target value histogram, count of instances in the node,preferred target value at the node, or a ranked list of target values atthe node.

An example of the application of a decision tree model is shown inFIG. 1. In this example, the decision tree models the response to acredit card promotion and may be used to provide a prediction as to theanswer to the question “Will a customer respond to a credit cardpromotion?” In order to obtain the prediction, information relating tothe particular customer may be used to traverse the tree by, at eachnode of the tree, using values of the customer's information to select abranch of the tree to follow. For example, the root of the tree, with noinformation about the customer, the prediction is that the customer is56% (150 Y, 120 N) likely to respond to the promotion. If the customer'sage is known, then if the age is greater than 30, the prediction is thatthe customer is 75% (135 Y, 35 N) likely to respond to the promotion. Ifthe age is less than or equal to 30, the prediction is that the customeris 15% (15 Y, 85 N) likely to respond to the promotion. If thecustomer's income is also known, then the prediction can be furtherrefined. If the customer's income is medium or low, then the predictionis that the customer is 3% (3 Y, 84 N) likely to respond to thepromotion. If the customer's income is high, then the prediction is thatthe customer is 92% (12 Y, 1 N) likely to respond to the promotion.Thus, although it may not be worthwhile to target the credit cardpromotion to people under 30 in general; targeting the promotion topeople under 30 with high incomes is worthwhile.

The present invention is particularly concerned with the generation of adecision tree model, such as that shown in FIG. 1. The present inventionimplements the functionality of generating a decision tree model in adatabase system. Preferably, the majority of the functionality isimplemented via an internal SQL table function leveraging parallelrecursion and bitmap indexes.

An exemplary data flow diagram of a process 200 of building a decisiontree model, including building and scoring of models and generation ofpredictions/recommendations, is shown in FIG. 2. The training/modelbuilding step 202 involves generating the decision tree models that maybe used to perform data mining predictions. The inputs to training/modelbuilding step 202 include training parameters 204, training data 206,and model building algorithms 208. Model building algorithms 208 includealgorithms that process the training data 206 in order to actually buildthe models. In particular, model building algorithms 208 includesdecision tree algorithms that are used to build data mining models thatare based on decision trees. Training parameters 204 are parameters thatare input to the data-mining model building algorithms to control howthe algorithms build the models. Training data 206 is data that is inputto the algorithms and which is used to actually build the models.

Training/model building step 202 invokes the data mining model buildingalgorithms included in model building algorithms 208, initializes thealgorithms using the training parameters 204, processes training data206 using the algorithms to build the model, and generates trained model210. Trained model 210 includes representations of the decision treemodel. Trained model 210 may also be evaluated and adjusted in order toimprove the quality, i.e. prediction accuracy, of the model. Trainedmodel 210 is then encoded in an appropriate format and deployed for usein making.

In the present invention, the bulk of the model building algorithms 208are implemented in the form of a new decision tree table function. Theinput to this function is training data 206 in the form of a set of rowscontaining predictors (like age, gender, etc.) and a categorical target(perhaps income_level). Each row contains all of the information for aparticular case. In addition, the table function has other inputs, suchas training parameters 204, to help guide the tree build process (e.g.,max tree depth).

A process 300 of in-database building of a decision tree model, such asthat performed in step 202 of FIG. 2, is shown in FIG. 3. Process 300begins with step 302, enumerate and feed, in which data is taken fromnormal rows in database tables and prepared for bitmap construction.

In step 304, the bitmaps are constructed. In order to construct thebitmaps, the incoming rows of data are numbered; then a bitmap isconstructed for each unique value of each predictor and target. Thesebitmaps indicatewhether or not that unique value of each predictor andtarget is present in that row. An example of this is shown in FIG. 4. Asshown in FIG. 4, a plurality of rows 401-409, etc., include a pluralityof values of predictors, such as age and income, as well as one or moretargets, such as their response to a promotion. A bitmap 450 isconstructed for age 1 that indicates whether or not the value 1 of thepredictor age is present in each row 401-409, etc. Likewise, bitmaps451-454 are constructed for other ages, and indicate whether or nottheir value of the predictor age is present in each row 401-409, etc. Inaddition, bitmaps for other predictors, such as income, etc., and forthe target, such as response, are constructed.

In step 306, the bitmaps are sorted by predictor and predictor value andtarget and target value, which may improve performance of the decisiontree generation process. In step 308, the sorted bitmaps are compacted,which also may improve performance of the decision tree generationprocess.

In step 310, once the compacted bitmaps are available, the counts ofpredictor-target pairs are generated. Preferably, this is done byintersecting a predictor bitmap with a target bitmap and counting theresulting bits. For example, the number of males with low income can becounted by intersecting the bitmaps for (gender, m) and (income_level,low) and counting the resulting bits—rows where both the predictor valueand target value are present.

In step 312 the resulting training data is ordered. Preferably, theordering depends upon the type of data being processed. For example, fornumerical data, the data is preferably ordered by predictor value, whilefor categorical data, the data is preferably ordered by target density.

In step 314, the counts generated in step 310 are used to determine,initially, for the root node, which predictor is the best splitter andwhere the split should occur. The splitting process of step 314 takesthe raw predictor-target counts (per node) and computes the best split,preferably using an impurity metric, such as the Gini impurity metric orthe entropy impurity metric. For example, the Gini impurity metric maybe defined as:

-   -   a. 1−SUM(p|t)ˆ2) over all target classes j,    -   b. p(j|t)=p(j,t)/p(t)=p(j,t)/SUM(p(j,t)),    -   c. p(j,t)=P(j)*Nj(t)/Nj,        where P(j) is the (altered) prior probability of class j, Nj(t)        is the number of records of class j in node t, and Nj is the        number of records of class j in whole training set.

It is to be noted that splitting considerations vary with the type ofdata to be split. For example, for Numerical predictors, possible splitpoints are along predictor value order (range splits). For categoricalpredictors with binary targets, possible split points lie along sortedorder of target density (class1cnt/(class1cnt+class2cnt)). Forcategorical predictors with multi-class targets, it is preferably to use“twoing”, that is, to arbitrarily group target classes into two “super”classes, use the regular approach for categoricals as above, andreassign targets to groups based on node dominance and repeat.

In step 316, the bitmaps for each child node generated by the split isgenerated. Once the best split is determined in step 314, the splitinformation is fed to step 316, so that the node bitmaps for the nextlevel can be generated. In addition, the best split information is sentto the pruning step 318 for further processing. The splitting step mayalso generate surrogate splits and target histograms, if desired.

Process 300 then loops back to step 310 in order to recursively performsteps 310-314 on each child node of the tree as the tree is split. Thetree is built in a breadth-first manner. First, the root split isdetermined. Once this is done, the root's two child node bitmaps aregenerated and the best splits for those two children are determined.Once this is done, the process moves to the third level, and so on.

It is to be noted that step 312 is among the steps that are repeated. Asdescribed above, the ordering performed by this step depends upon thetype of data being processed. For example, for numerical data, the datais preferably ordered by predictor value, while for categorical data,the data is preferably ordered by target density. Each split point alongthe sorted path is evaluated using an impurity metric. The best splitpoint determined this way is preserved and compared to the previous bestpredictor split. When the process has finished with a set of nodes, itreturns the best splits found.

In step 318 the tree is pruned by walking the decision tree and using aMinimum Description Length (MDL) based pruning approach to trim offleaves and branches. The pruned tree is then output from process 300.The main purpose of pruning is to take the built tree and prune so thatit is general (not over-trained). In addition, during the pruning phasenodes are renumbered so that branch nodes start with 0 and arecontiguous and extra splits and surrogates are eliminated. Inputs to thepruning process include rows of data that are outputs from the buildprocess, using an encoding. These basic rows are

Class total rows (node target histogram)

Main split

Surrogate splits

Special rows are:

Split predictor cardinality (for split cost)

Binning rows (to unmap bin values)

Predictor counts (for split cost)or

Target class cardinality (for node cost)

In order to produce a split for a given predictor of a given node andprovide a measure of “goodness” for the split, it is preferred that asingle process have all of the predictor-target counts for thatpredictor for that node. This is not strictly necessary, but reducesimplementation complexity significantly.

An exemplary interface defining an SQL statement that invokesin-database generation of a decision tree model is shown in FIG. 5. TheSQL statement defined by this interface is labeledORA_FI_DECISION_TREE_HORIZ. An example of the use of this statement inSQL code is shown in FIG. 6. Typically, users would invoke the SQL codeshown in FIG. 6 through a PL/SQL API, an example of which is shown inFIG. 7.

An exemplary block diagram of a database system 800, in which thepresent invention may be implemented, is shown in FIG. 8. Databasesystem 800 is typically a programmed general-purpose computer system,such as a personal computer, workstation, server system, andminicomputer or mainframe computer. Database system 800 includes one ormore processors (CPUs) 802A-802N, input/output circuitry 804, networkadapter 806, and memory 808. CPUs 802A-802N execute program instructionsin order to carry out the functions of the present invention. Typically,CPUs 802A-802N are one or more microprocessors, such as an INTELPENTIUM® processor. FIG. 8 illustrates an embodiment in which databasesystem 800 is implemented as a single multi-processor computer system,in which multiple processors 802A-802N share system resources, such asmemory 808, input/output circuitry 804, and network adapter 806.However, the present invention also contemplates embodiments in whichdatabase system 800 is implemented as a plurality of networked computersystems, which may be single-processor computer systems, multi-processorcomputer systems, or a mix thereof.

Input/output circuitry 804 provides the capability to input data to, oroutput data from, database system 800, For example, input/outputcircuitry may include input devices, such as keyboards, mice, touchpads,trackballs, scanners, etc., output devices, such as video adapters,monitors, printers, etc., and input/output devices, such as, modems,etc. Network adapter 806 interfaces database system 800 withInternet/intranet 810. Internet/intranet 800 may include one or morestandard local area network (LAN) or wide area network (WAN), such asEthernet, Token Ring, the Internet, or a private or proprietary LAN/WAN.

Memory 808 stores program instructions that are executed by, and datathat are used and processed by, CPU 802 to perform the functions ofdatabase system 800. Memory 808 may include electronic memory devices,such as random-access memory (RAM), read-only memory (ROM), programmableread-only memory (PROM), electrically erasable programmable read-onlymemory (EEPROM), flash memory, etc., and electro-mechanical memory, suchas magnetic disk drives, tape drives, optical disk drives, etc., whichmay use an integrated drive electronics (IDE) interface, or a variationor enhancement thereof, such as enhanced IDE (EIDE) or ultra directmemory access (UDMA), or a small computer system interface (SCSI) basedinterface, or a variation or enhancement thereof, such as fast-SCSI,wide-SCSI, fast and wide-SCSI, etc, or a fiber channel-arbitrated loop(FC-AL) interface.

In the example shown in FIG. 8, memory 808 includes compilationcomponent routines 812, counting component routines 814, splittingcomponent routines 816, pruning component routines 818, persistingcomponent routines 820, viewing component routines 822, training data824, decision tree model 826, and operating system 828. Compilationcomponent routines 812 compile the SQL table function. Countingcomponent routines 814 perform the enumerate and feed functions, inwhich data is taken from normal rows in database tables and prepared forbitmap construction, generate the bitmaps (predictor, target, and node),intersect the bitmaps, and count the results. Splitting componentroutines 816 find the best split and surrogates for each node. Pruningcomponent routines 818 prune the resulting tree. Persisting componentroutines take the output of the table function and produce a data miningdecision tree model 826. Viewing component routines 822 take a builtmodel and return its details. Training data 824 is data used by theroutines to generate the decision tree model. Operating system 828provides overall system functionality.

As shown in FIG. 8, the present invention contemplates implementation ona system or systems that provide multi-processor, multi-tasking,multi-process, and/or multi-thread computing, as well as implementationon systems that provide only single processor, single thread computing.Multi-processor computing involves performing computing using more thanone processor. Multi-tasking computing involves performing computingusing more than one operating system task. A task is an operating systemconcept that refers to the combination of a program being executed andbookkeeping information used by the operating system. Whenever a programis executed, the operating system creates a new task for it. The task islike an envelope for the program in that it identifies the program witha task number and attaches other bookkeeping information to it. Manyoperating systems, including UNIX®, OS/2®, and Windows®, are capable ofrunning many tasks at the same time and are called multitaskingoperating systems. Multi-tasking is the ability of an operating systemto execute more than one executable at the same time. Each executable isrunning in its own address space, meaning that the executables have noway to share any of their memory. This has advantages, because it isimpossible for any program to damage the execution of any of the otherprograms running on the system. However, the programs have no way toexchange any information except through the operating system (or byreading files stored on the file system). Multi-process computing issimilar to multi-tasking computing, as the terms task and process areoften used interchangeably, although some operating systems make adistinction between the two.

Additional processing 900 relating to counting of predictor-target pairsand generation of count tables including these counts are shown in FIG.9. The present invention provides a bitmap-based technique forconstructing the count tables. The count tables may be constructed inparallel. The present invention supports out-of-memory processing whennot all the bitmaps and count tables can fit in memory. Process 900begins with step 902, in which the split node information is read andthe best split point is found. For example, for numerical data, a splitfrom node-0 might be determined as:

-   (node-0, age <=35, node-1)-   (node-0, age >35, node-2),    while for categorical data, a split might be determined as:-   (node-1, education in (‘master’, ‘PhD’), node-5)-   (node-1, education in (‘bachelor’, ‘else’), node-6).

In step 904, bitmaps for the split nodes are generated by intersectingthe bitmap of the parent node and the bitmaps of the predictor thatsatisfy the condition of that node to generated the new split nodebitmaps. For example, for numerical data, a bitmap may be generated as:

-   (node-0, age <=35, node-1)-   Bitmap of node-1: intersect the bitmap of node-0 with the result of    ORing all <age> predictor bitmaps whose key is less than or equal to    35,    while for categorical data, a bitmap may be generated as:-   (node-1, education in (‘master’, ‘PhD’), node-5)-   Bitmap of node-5: intersect the bitmap of node-1 with the result or    ORing education[‘master’]'s bitmap and education[‘PhD’]'s bitmap

In step 906, the bit count of the intersected bitmaps may be generatedby counting the bits that are set in the resulting bitmap that wasgenerated by intersecting the split node bitmaps with the predictorbitmaps and with the target bitmaps.

Thus, the count of a predictor-target pair may be computed byintersecting a predictor bitmap with a target bitmap and counting thenumber of bits set to 1 in the result. For example, the process cancount the number of males with low income by intersecting the bitmapsfor (gender, male) and (income_level, low) and counting the number ofbits set to 1 in the result.

Once the processing is below the root node, it needs to compute thesepredictor-target counts with respect to the rows that fall into thatnode (by following the splitters in the tree). In order to do thisnode-local computation, the process will create node bitmaps by ORingand ANDing predictor bitmaps based on the tree splits. For example, ifthe root split was decided to be age <=20, then the process wouldproduce a node bitmap by ORing all of the bitmaps for age where thevalue is <=20 and another node bitmap by ORing the values of age >20. Ata lower level, the process would AND the age bitmap with the lower levelsplitter OR'd bitmap. Once the process has node bitmaps, it just needsto intersect these bitmaps with the predictor-target intersections toget the per node predictor-target counts.

As mentioned above, the present invention supports out-of-memoryprocessing when not all the bitmaps and count tables can fit in memory.An example of processing where all the bitmaps and count tables do fitin memory is shown in FIG. 10. In this case, it is simply a matter ofintersecting the target bitmaps with the split node bitmaps, thenintersecting that result with the predictor bitmaps. When the bitmapsand count tables are too large to fit in memory, additional processingis necessary. For example, in FIG. 11, a case where a certain portion ofthe bitmaps will fit in memory is shown. In this case, the number ofbitmaps that will fit in memory is T*N bitmaps, where T the number ofthe target bitmaps and N is the number of the split node bitmaps. Whenthe T target bitmaps are intersected with the N split node bitmaps, theresult is a bitmap array including T*N bitmaps. Each of the T*N bitmapsis then intersected with each of the predictor bitmaps to yield theresult.

Then T*N bitmaps will not fit in memory, then processing similar to thatshown in FIG. 12 may be used. In the example of FIG. 12, the T targetbitmaps are intersected with a portion of the split node bitmaps (N/P)to generated T*N/P bitmaps, which will fit in memory. Each of the T*N/Pbitmaps is then intersected with each of the predictor bitmaps to yielda partial result. The T target bitmaps are then intersected with thenext set of N/P split node bitmaps, and so on.

It is important to note that while the present invention has beendescribed in the context of a fully functioning data processing system,those of ordinary skill in the art will appreciate that the processes ofthe present invention are capable of being distributed in the form of acomputer readable medium of instructions and a variety of forms and thatthe present invention applies equally regardless of the particular typeof signal bearing media actually used to carry out the distribution.Examples of computer readable media include recordable-type media suchas floppy disc, a hard disk drive, RAM, and CD-ROM's, as well astransmission-type media, such as digital and analog communicationslinks.

Although specific embodiments of the present invention have beendescribed, it will be understood by those of skill in the art that thereare other embodiments that are equivalent to the described embodiments.Accordingly, it is to be understood that the invention is not to belimited by the specific illustrated embodiments, but only by the scopeof the appended claims.

1. A computer-implemented method of counting predictor-target pairs for a decision tree model, the decision tree model based on data stored in a database, the data comprising a plurality of rows of data, at least one predictor and at least one target, the method comprising: generating a split node bitmap for each tree node corresponding to a subset of data stored in a database system by intersecting a parent node bitmap and a bitmap of a predictor that satisfies a condition of the node; intersecting each split node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; and counting bits of each intersected bitmap to generate a count of per-node predictor-target pairs.
 2. The method of claim 1, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: intersecting each target bitmap with each split node bitmap to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form an intersected bitmap.
 3. The method of claim 2, wherein the target bitmaps and the split node bitmaps fit in a memory of a computer.
 4. The method of claim 1, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: for each of a plurality of portions of the split node bitmaps: intersecting each target bitmap with each split node bitmap in the portion of the split node bitmaps to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form the intersected bitmaps.
 5. The method of claim 4, wherein the target bitmaps and a portion of the split node bitmaps fit in a memory of a computer.
 6. A method of generating a decision tree model comprising: generating a plurality of bitmaps in the database system, the bitmaps generated from data stored in a database table in the database system, the database table comprising a plurality of rows of data, the plurality of bitmaps comprising a bitmap for each unique value of each predictor and target and indicating whether or not that unique value of each predictor and target is present in each row of the database table; intersecting each split node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; counting bits of each intersected bitmap to generate a count of predictor-target pairs; determining a splitter value for the data in the database table using the counts of the predictor-target pairs so as to split the data in the database table into a plurality of child nodes, each child node comprising a portion of the data in the database table; generating child bitmaps for the data in each child node; recursively generating a split node bitmap for each child node by intersecting a parent node bitmap and a bitmap of a predictor that satisfies a condition of the child node; intersecting each child node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; and counting bits of each intersected bitmap to generate a count of predictor-target pairs; whereby a decision tree model is formed.
 7. The method of claim 6, wherein the bitmaps are sorted by predictor and predictor value and target and target value.
 8. The method of claim 6, wherein the bitmap that satisfies the condition of the node is generated by: if the node is a root node, generating a bitmap by ORing each bitmap for each value of the predictor that satisfies a condition of the node split, to form a single bitmap for all values of the predictor that satisfy the condition of the node split; and if the node is below the root node, generating a bitmap by ORing each bitmap for each value of the predictor that satisfies a condition of the node split, to form a single bitmap for all values of the predictor that satisfy the condition of the node split and ANDing the single bitmap with a bitmap for a node above the node.
 9. The method of claim 8, wherein the bitmaps are sorted by predictor and predictor value and target and target value.
 10. A database system for counting predictor-target pairs for a decision tree model, the decision tree model based on data stored in a database, the data comprising a plurality of rows of data, at least one predictor and at least one target, the system comprising: a processor operable to execute computer program instructions; a memory operable to store computer program instructions executable by the processor; and computer program instructions stored in the memory and executable to perform the steps of: generating a split node bitmap for each split node of data stored in a database system by intersecting a parent node bitmap and a bitmap of a predictor that satisfies a condition of the node; intersecting each split node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; and counting bits of each intersected bitmap to generate a count of predictor-target pairs.
 11. The system of claim 10, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: intersecting each target bitmap with each split node bitmap to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form an intersected bitmap.
 12. The system of claim 11, wherein the target bitmaps and the split node bitmaps fit in a memory of a computer.
 13. The system of claim 10, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: for each of a plurality of portions of the split node bitmaps: intersecting each target bitmap with each split node bitmap in the portion of the split node bitmaps to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form the intersected bitmaps.
 14. The system of claim 13, wherein the target bitmaps and a portion of the split node bitmaps fit in a memory of a computer.
 15. A database system for generating a decision tree model comprising: a processor operable to execute computer program instructions; a memory operable to store computer program instructions executable by the processor; and computer program instructions stored in the memory and executable to perform the steps of: generating a plurality of bitmaps in the database system, the bitmaps generated from data stored in a database table in the database system, the database table comprising a plurality of rows of data, the plurality of bitmaps comprising a bitmap for each unique value of each predictor and target and indicating whether or not that unique value of each predictor and target is present in each row of the database table; intersecting each split node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; counting bits of each intersected bitmap to generate a count of predictor-target pairs; determining a splitter value for the data in the database table using the counts of the predictor-target pairs so as to split the data in the database table into a plurality of child nodes, each child node comprising a portion of the data in the database table; generating child bitmaps for the data in each child node; recursively generating a bitmap for each child node by intersecting a parent node bitmap and a bitmap of a predictor that satisfies a condition of the child node; intersecting each child node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; and counting bits of each intersected bitmap to generate a count of predictor-target pairs; whereby a decision tree model is formed.
 16. The system of claim 15, wherein the bitmaps are sorted by predictor and predictor value and target and target value.
 17. The system of claim 15, wherein the bitmap that satisfies the condition of the node is generated by: if the node is a root node, generating a bitmap by ORing each bitmap for each value of the predictor that satisfies a condition of the node split, to form a single bitmap for all values of the predictor that satisfy the condition of the node split; and if the node is below the root node, generating a bitmap by ORing each bitmap for each value of the predictor that satisfies a condition of the node split, to form a single bitmap for all values of the predictor that satisfy the condition of the node split and ANDing the single bitmap with a bitmap for a node above the node.
 18. The system of claim 17, wherein the bitmaps are sorted by predictor and predictor value and target and target value.
 19. A computer program product for counting predictor-target pairs for a decision tree model, the decision tree model based on data stored in a database, the data comprising a plurality of rows of data, at least one predictor and at least one target, the computer program product comprising: a computer readable medium; computer program instructions, recorded on the computer readable medium, executable by a processor, for performing the steps of: generating a split node bitmap for each split node of data stored in a database system by intersecting a parent node bitmap and a bitmap of a predictor that satisfies a condition of the node; intersecting each split node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; and counting bits of each intersected bitmap to generate a count of predictor-target pairs.
 20. The computer program product of claim 19, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: intersecting each target bitmap with each split node bitmap to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form an intersected bitmap.
 21. The computer program product of claim 20, wherein the target bitmaps and the split node bitmaps fit in the memory of a computer.
 22. The computer program product of claim 19, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: for each of a plurality of portions of the split node bitmaps: intersecting each target bitmap with each split node bitmap in the portion of the split node bitmaps to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form the intersected bitmaps.
 23. The computer program product of claim 22, wherein the target bitmaps and a portion of the split node bitmaps fit in a memory of a computer.
 24. A computer program product for generating a decision tree model in a database system, comprising: a computer readable medium; computer program instructions, recorded on the computer readable medium, executable by a processor, for performing the steps of generating a plurality of bitmaps in the database system, the bitmaps generated from data stored in a database table in the database system, the database table comprising a plurality of rows of data, the plurality of bitmaps comprising a bitmap for each unique value of each predictor and target and indicating whether or not that unique value of each predictor and target is present in each row of the database table; intersecting each split node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; counting bits of each intersected bitmap to generate a count of predictor-target pairs; determining a splitter value for the data in the database table using the counts of the predictor-target pairs so as to split the data in the database table into a plurality of child nodes, each child node comprising a portion of the data in the database table; generating child bitmaps for the data in each child node; recursively generating a bitmap for each child node by intersecting a parent node bitmap and a bitmap of a predictor that satisfies a condition of the child node; intersecting each child node bitmap with each predictor bitmap and with each target bitmap to form intersected bitmaps; and counting bits of each intersected bitmap to generate a count of predictor-target pairs; whereby a decision tree model is formed.
 25. The computer program product of claim 24, wherein the bitmaps are sorted by predictor and predictor value and target and target value.
 26. The computer program product of claim 24, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: intersecting each target bitmap with each split node bitmap to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form an intersected bitmap.
 27. The computer program product of claim 26, wherein the target bitmaps and the split node bitmaps fit in a memory of a computer.
 28. The computer program product of claim 24, wherein each split node bitmap is intersected with each predictor bitmap and with each target bitmap to form intersected bitmaps by: for each of a plurality of portions of the split node bitmaps: intersecting each target bitmap with each split node bitmap in the portion of the split node bitmaps to form a plurality of intermediate bitmaps; and intersecting each intermediate bitmap with each predictor bitmap to form the intersected bitmaps.
 29. The computer program product of claim 28, wherein the target bitmaps and a portion of the split node bitmaps fit in a memory of a computer.
 30. The computer program product of claim 24, wherein the bitmap that satisfies the condition of the node is generated by: if the node is a root node, generating a bitmap by ORing each bitmap for each value of the predictor that satisfies a condition of the node split, to form a single bitmap for all values of the predictor that satisfy the condition of the node split; and if the node is below the root node, generating a bitmap by ORing each bitmap for each value of the predictor that satisfies a condition of the node split, to form a single bitmap for all values of the predictor that satisfy the condition of the node split and ANDing the single bitmap with a bitmap for a node above the node.
 31. The computer program product of claim 30, wherein the bitmaps are sorted by predictor and predictor value and target and target value. 